Quality and Schools 


Looking at School Performance From a Different Perspective: 
A Maryland Report Card Reflecting Students Served 


Christopher Wohn, M.D., Liepa Bobenene, Ph.D. 


June 2019 


Introduction 


The federal Every Student Succeeds Act (ESSA) that passed in 2015 changed many aspects of school 
accountability across the United States. Under ESSA states had to interpret new requirements and create 
accountability plans. Maryland developed a plan that was approved by the U.S. Department of Education in 
2018. As part of Maryland’s ESSA plan, a new school report, the Maryland Report Card, was developed to 
evaluate school performance across multiple metrics, including assessment performance, student growth, school 
climate, and implementation of a well-rounded curriculum. Schools across the state were ranked according to 
how many points they earned by meeting performance expectations on these indicators, and each school 
received a summative “star” rating, from one to five stars. 


Schools are complex entities and any measurement of overall school performance can be attributed to multiple 
inputs: quality of teachers, socio-economic background of students, involvement of parents in the school 
community, resources of the governing school district, and so on. Summative school evaluation systems, such as 
the Maryland Report Card, often take a blunt approach to judging school success and can fail to recognize 
outstanding instruction and academic gains in school communities that exist in high poverty. Accountability 
metrics with no consideration of students served may end up in large part representing the socio-economic level 
of students’ families rather than the quality of each school. 


Socio-economic Measures in ESSA. Economic disadvantage (ED) is the term used in ESSA, and states must 
determine their own definitions. In Maryland, ED is measured by counting students who are directly certified, 
which means they are enrolled in the Supplemental Nutrition Assistance Program (SNAP), Temporary 
Assistance for Needy Families (TANF), federal foster care, or they are experiencing homelessness. When the 
percentage of points earned on the Maryland Report Card is visualized against the percentage of students that 
are identified as economically disadvantaged at a school, the correlation between Report Card performance and 
poverty can be easily appreciated (Figure 1). 
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Figure 1. Elementary school report card performance compared to student economic disadvantage. 


baltimore-berc.org |1 


Quality and Schools 


The relationship between academic achievement and poverty is one of the most consistent findings in education 
research, and the income-achievement gap has been growing for at least 50 years. Based on Maryland Report 
Card results, we see that as a school’s population becomes more concentrated in poverty, the number of points it 
earns on the Report Card decreases. This trend plays out in star rating as well, with high poverty elementary 
schools (> 60% ED) never earning a five-star rating. To some extent, the Maryland Report Card results reinforce 
the influence of poverty rather than capturing the value added by educators in light of the different student 
populations that schools serve. 


Data Sourcesand Methods 


All report card data are publicly available and were obtained from the Maryland State Department of Education 
(MSDE) in April 2019, reflecting any data updates captured through the state’s rating appeal process. The 
Maryland Report Cards released in December 2018 were based on school year 2017-18 data. Student economic 
disadvantage (quantified as direct certification) and enrollment counts were obtained from the MSDE Food and 
Nutrition website for the 2017-18 school year. For this analysis, schools spanning grade bands, e.g., schools 
serving grades K-8, are reported as both elementary (including data for grades K-5) and middle (including 
grades 6-8), rather than combining to a summative rating for the whole school as was done in the Maryland 
Report Card. 


Borrowing from a different field, we estimate the relationship between school economic disadvantage rate and 
the percentage of Report Card points earned by schools across the state using quantile regression. Quantile 
regression is the technique used to estimate height and weight percentile charts for infants and children (Figure 
2). It is also used to calculate student growth percentiles (SGP), which are a part of the academic progress 
indicator in the Maryland Report Card. 

Birth to 36 months: Boys 


y' NAME 
Length-for-age and Weight-for-age percentiles RECORD # 
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\ 


Be 


x40zmr 


Z 
Z 
Fee 
“7 
Ss 


se 
ge 
| 
eeagEESS 


Wi 
(fee 
eccee 


4azrO-mse 


Acton ty 20220 (roates 4200") (ape 
SOC Oovoubad ty a Nara Crk te ea Sum terse ah 
re tor tr . 
Moo wemcds gon owner aren neaarnienr ror 


Figure 2: Pediatric growth charts with quantile regression. 
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All schools with available data on the Maryland Report Card website were included. The quantiles chosen were 
those associated with each of the star ratings, based on decisions from the November 2018 MSDE Board 
Meeting (Option 2c). 


Table 1. Maryland Report Card Quantiles 
Range of Points Earned 
(%) 

0-30 

30-45 

45-60 

60-75 

5 75-100 
Source: Maryland State Department of Education, 2018 
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These Report Card quantiles were used to calculate new star ratings adjusted for students served by schools 
through the steps outlined below (Figure 3). Similar to how percentiles of weight and height are estimated by 
age on a pediatric growth chart, adjusted Report Card score quantiles were estimated as a function of economic 
disadvantage for schools in Maryland. This method ensures that adjusted results have the same number of 
schools in each star rating category for each grade band. The percentile ranks associated with each star rating 
are provided in the Appendix. 


Star rating cutoffs were 
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were regressed on new star ratings based on 
Economic Disadvantage regressed cutoffs 


Figure 3: Stepsin determining corelation-adjusted star ratings. 


Results 


New star ratings were assigned to schools based on the regression procedure described above. This resulted in a 
banding of star rating that adjusts for the estimated correlation between percentage of Report Card points 
earned and school economic disadvantage rates. Figure 4 represents original and adjusted star ratings (color 
coded) for elementary school grade bands in Maryland. These plots illustrate the effect of adjusting for the ED- 
Report Card points correlation among elementary schools. Many schools with more than 60% of students in 
economic disadvantage received five stars after including economic disadvantage (see right-hand panel, as 
opposed to the left-hand panel of Figure 4). 
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Figure 4. Orginal star ratings (left) compared to ratings adjusted for student economic disadvantage 


(right). 


This adjustment provides high poverty schools an opportunity to be recognized for the value they add to high- 
need student populations, rather than getting scores that reflect the socio-economic background of students. 
Three high poverty schools are highlighted by shapes (triangle, oval and square) in Figure 4 to allow readers to 
focus on the changes in star ratings created by the adjustment. Click here for a complete list of all Maryland 
schools with their original star ratings, and the new correlation-adjusted star ratings. Click here for summary 
sheets of elementary, middle and high schools. Table 2 includes the average star rating for each Local Education 
Agency (LEA) before and after the adjustment. 


Disc ussion 


As shown in Table 2, changes in average star ratings for districts are as large as a full star. In Baltimore City, 
one of the highest poverty districts in Maryland, the number of five-star schools increased dramatically after 
adjusting for economic disadvantage. In the original ratings, only two high schools were identified as five-star 
schools; in the adjusted ratings, twelve received five-star ratings. The size of the change reinforces the finding 
that concentrated poverty has an enormous impact on the ratings released in the Maryland Report Cards. 


This leads us to consider the purpose of the accountability requirements under ESSA and how the data 
reporting can help achieve the goals of the legislation. Should the star ratings released by the Maryland State 
Department of Education reflect the characteristics of students enrolled at each school, or the effectiveness of 
the educators given different school contexts? And how can these ratings be used for improvement? 


State Superintendent of Schools Karen Salmon highlights the intent to improve schools through the Maryland 
Report Cards, to set “our schools on the path to continuous improvement,” with the ultimate goal of ensuring 
“ALL students have access to the opportunity to attain an excellent education” (Salmon, 2018). Low star ratings 
can translate into additional funding support for schools identified for improvement. Yet, a low star rating may 
in fact negatively impact schools that already struggle in the most challenging circumstances. Schools may be 
harmed by a loss of enrolled students, high quality teachers and administrators, or community support 
following the public communication of a low star rating. 
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Table 2 
Rating Summary by Local Education Agency (LEA) of Original Star Ratings and 
Ratings Adjusted for Student Economic Disadvantage 


Average Average Change in 
Original Adjusted Average 
Star Rating Star Rating Rating 
Allegany 4.1 4.6 0.5 
Anne Arundel 3.1 2.3 -0.8 
Baltimore City BH 3.6 ila 
Baltimore County 3.6 3.6 0 
Calvert 4.3 3.8 -0.5 
Caroline 4 4.4 0.4 
Carroll 4.4 4.1 -0.3 
Cecil 3.7 3.9 0.2 
Charles 3.8 3.6 -0.2 
Dorchester 3.6 4.2 0.6 
Frederick 4.1 3.6 -0.5 
Garrett 4.1 4.2 0.1 
Harford 3.8 3.5 -0.3 
Howard 4.3 3.9 -0.4 
Kent 3.8 4.2 0.4 
Montgomery 4 3.7 -0.3 
Prince George's 3.4 3.1 -0.3 
Queen Anne's 4.3 4.3 0 
Saint Mary's 4.1 4 -0.1 
Somerset 3.3 4.3 1 
Talbot 4 Al 0.1 
Washington 4.2 4.2 0 
Wicomico 3.6 4.3 0.7 
Worcester 4 4.3 0.3 


There is evidence that families, particularly those with high incomes, make housing decisions based on school 
ratings that end up increasing segregation. A recent review of the effects of “Great Schools” ratings across the 
U.S. from 2006 — 2015 found that the availability of school ratings accelerated disparities in housing values and 
income distributions across communities, as neighborhoods with schools with low ratings lost high-income and 
college-educated residents. Therefore, the availability of school ratings resulted in less school integration and 
less equity in education rather than more (Hasan & Kumar, 2018). 


Given that a substantial proportion of the Maryland Report Card star rating is explained by the percentage of 
students in economic disadvantage, it seems that a strategy to improve ratings would be to reduce family 
poverty or attract more affluent families into a school community. Yet, a “1 star” rating may make the latter 
impossible or even cause an increase in concentrated poverty. Replacing staff at schools that fail to improve or 
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closing them completely may not benefit our poorest students unless these students are redirected to schools 
that are much higher performing (Berner & Steiner, 2019). The long-term effects of being labeled a “1-star” 
school should be analyzed in future years, tracking the cohorts of students currently attending these schools. 


The method used in this brief is one of many that could help account for differences in student populations when 
rating school performance. A simpler approach of grouping and ranking schools according to economic 
disadvantage rates could also address the issue. In addition to economic disadvantage, other characteristics of 
the student population, such as disability or English learner status, could be considered to create more fair 
comparisons between schools. Statistical techniques such as propensity score matching would also offer 
opportunities to create even more accurate comparison groups for schools and students. See a complete list of all 


Maryland schools, a summary sheet for elementary, middle and high schools. 


Conclusion 


This brief explores how school rating systems can represent the effectiveness of educators as opposed to the 
characteristics of their students. When Maryland Report Card ratings are adjusted for economic disadvantage, 
it helps us better understand school quality in terms of the value that educators bring to schools, as opposed to 
the challenges and advantages that students themselves bring. Using the new star ratings adjusted for 
economic disadvantage, we can identify schools that are making strides in the highest concentrations of poverty 
and potentially avoid the negative consequences associated with public communication of low star ratings for 
high poverty schools that are making gains. This may better help some of our most vulnerable students close 
gaps in both educational access and outcomes. 
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Appendices 


AppendixA 


Percentile Ranks of Points Eamed on Maryland Report Card by Grade Band 


Gta Rats Elementary Percentile Rank Middle Percentile High Percentile Rank 
Cutoffs Rank Cutoffs Cutoffs 
1 0-2.4 0-4.4 0-3.3 
2 2.5-12.8 4.5-20.5 3.4-19.8 
3 12.9-36.4 20.6-57.4 19.9-44.1 
4 36.5-82.7 57.5-90.5 44.2-80.3 
5 82.8-100 90.6-100 80.4-100 
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